Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an IChatClient implementation to OnnxRuntimeGenAI #987

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

stephentoub
Copy link
Member

No description provided.

src/csharp/ChatClient.cs Outdated Show resolved Hide resolved
src/csharp/ChatClient.cs Outdated Show resolved Hide resolved
Copy link
Contributor

@azchohfi azchohfi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then only issue I found is that when ChatOptions is null, it fails for Phi 3.5. This works fine for Phi3.
Phi 3.5 requires max_length to be provided. Seems to be a bug on Phi3.5, but I do not see a way around it, other than always providing max_length.

@natke
Copy link
Contributor

natke commented Nov 5, 2024

@stephentoub can you please merge main into your branch? This should enable the CIs to pass

@stephentoub
Copy link
Member Author

@stephentoub can you please merge main into your branch? This should enable the CIs to pass

Thanks, but it's already based off of the latest commit in main. It seems like whatever feed is being used may not contain some of the necessary packages, like System.Text.Json?

@pranavsharma
Copy link
Contributor

Can you add a test?

@azchohfi
Copy link
Contributor

@pranavsharma Seems like the package needs to be added to the feed, from the upstream (nuget.org).

@stephentoub
Copy link
Member Author

Can you add a test?

Yes

@azchohfi
Copy link
Contributor

Build is still failing due to the package not being available on the ORT-Nightly feed.
It needs to be manually added. @pranavsharma I do not have access to do that.

@stephentoub
Copy link
Member Author

@pranavsharma, I will update the PR with tests, but I first need your help in getting the feed updated with the necessary dependencies. Otherwise, nothing here even builds.

@baijumeswani
Copy link
Collaborator

I can help with that. Is System.Text.Json the only package needed. Will try to have it resolved soon.

@stephentoub
Copy link
Member Author

stephentoub commented Nov 25, 2024

I can help with that. Is System.Text.Json the only package needed. Will try to have it resolved soon.

Thanks. I'm not sure what's currently available in the feed, but at a minimum you'd need to add Microsoft.Extensions.AI.Abstractions. That itself has a System.Text.Json dependency downlevel.

@stephentoub
Copy link
Member Author

@baijumeswani, also #970 needs Microsoft.ML.Tokenizers

@baijumeswani
Copy link
Collaborator

Thanks for the info. A few of our engineering systems team members are out of office this week. And I am trying to find out how to upload nuget packages to our feed. It might take until next week to have this setup.

@baijumeswani
Copy link
Collaborator

I have (hopefully) added all the dependencies. Please see the latest windows cpu build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants